Recent developments in differentially private (DP) machine learning and DP Bayesian learning have enabled learning under strong privacy guarantees for the training data subjects. In this paper, we further extend the applicability of DP Bayesian learning by presenting the first general DP Markov chain Monte Carlo (MCMC) algorithm whose privacy-guarantees are not subject to unrealistic assumptions on Markov chain convergence and that is applicable to posterior inference in arbitrary models. Our algorithm is based on a decomposition of the Barker acceptance test that allows evaluating the Rényi DP privacy cost of the accept-reject choice. We further show how to improve the DP guarantee through data subsampling and approximate acceptance tests.

This work provides a detailed Renyi DP analysis of a modified MCMC acceptance test, and empirically demonstrates its efficacy. Originality: the RDP analysis and modified acceptance test is a novel contribution. Quality: the work is a complete piece on exploring this MCMC method, with a detailed analysis and experiments. Clarity: the work is fairly clearly written, but it can be easy to lose track of exactly what parameters remain as choices to be tuned in a list of various corrective factors and approximations. Significance: the work gives an MCMC method with privacy without convergence, which permits privacy guarantees to be given over a multitude of problems without doubts or guess work about when to stop the chain.

Although the analysis follows some known ideas from the literature on private SGD, there are a number of new tricks which make the current approach interesting, most notably the observation that one can use randomized acceptance tests to preserve privacy in an MCMC algorithm. When preparing the final version of this manuscript the authors should carefully consider the points raised in the reviews regarding: clarifying where the contributions lie with respect to previous work; provide high-level intuitions of the proofs to help a reader navigate the derivations; discuss the role of approximations used in the paper, where they affect the privacy or utility of the method, and where there is some room left for improvement.

Recent developments in differentially private (DP) machine learning and DP Bayesian learning have enabled learning under strong privacy guarantees for the training data subjects. In this paper, we further extend the applicability of DP Bayesian learning by presenting the first general DP Markov chain Monte Carlo (MCMC) algorithm whose privacy-guarantees are not subject to unrealistic assumptions on Markov chain convergence and that is applicable to posterior inference in arbitrary models. Our algorithm is based on a decomposition of the Barker acceptance test that allows evaluating the Rényi DP privacy cost of the accept-reject choice. We further show how to improve the DP guarantee through data subsampling and approximate acceptance tests.

Recent developments in differentially private (DP) machine learning and DP Bayesian learning have enabled learning under strong privacy guarantees for the training data subjects. In this paper, we further extend the applicability of DP Bayesian learning by presenting the first general DP Markov chain Monte Carlo (MCMC) algorithm whose privacy-guarantees are not subject to unrealistic assumptions on Markov chain convergence and that is applicable to posterior inference in arbitrary models. Our algorithm is based on a decomposition of the Barker acceptance test that allows evaluating the Rényi DP privacy cost of the accept-reject choice. We further show how to improve the DP guarantee through data subsampling and approximate acceptance tests. Papers published at the Neural Information Processing Systems Conference.